ITF found in Science mode (LN3_SQZ)
22:50 UTC a drop in the Horizon down at 35 Mpc (TBC), plot attached. I exit from Science for one minute.
https://gracedb.ligo.org/superevents/S240512r/view/
DAQ
FCEM_MotSwitch VPM process crashed twice. Properly restarted at 21:02, 21:15 UTC.
Guard Tour (UTC):
21:02 - 21:30
23:13 - 23:41
01:06 - 01:35
03:01 - 03:33
ITF found in Science mode (LN3_SQZ).
At 17:33 UTC ITF unlocked (TBC). After the unlock the DET_MAIN metatron node got stuck in UNKNOWN state. INIT required.
Relocked at first attempt. Science Mode set at 18:28 UTC.
No DMS Events to report
Guard tours (UTC)
17:02 - 17:31
19:11 - 19:38
ITF found in Science mode (LN3_SQZ)
16:00 UTC - CALIBRATION MODE SET
Planned weekly calibrations started:
- 16:01 UTC CALIBRATED_DF_PCAL (OK)
- 16:33 UTC CALIBRATED_DF_SENSITIVITY (OK)
Manual Unlock at 17:30 UTC
- 17:31 UTC CALIBRATED_DELAY (OK)
- 17:38 UTC CALIBRATED_PRWI (OK)
- 18:25 UTC CALIBRATED_DF_INtoEND_LN (OK)
ITF back in Science Mode at 19:59 UTC
Guard tour (UTC)
17:02 - 17:28
19:01 - 19:30
ITF stable locked at LOW_NOISE_3_SQZ (Science Mode) for the whole shift.
ITF found in Science Mode (LN3_SQZ).
At 02:53 UTC ITF unlocked for unknown reasons (TBC). Relocked at first attempt at 03:43 UTC. Science Mode set.
Guard Tour (UTC):
21:30 - 22:00
23:52 - 00:23
02:00 - 02:30
03:25 - 03:55
I found ITF in SCIENCE MODE.
It remained locked for the whole shift except for one unlock, at 21:22 UTC, done by mistake.
Guard Tour (UTC):
21:06 - 21:36
23:05 - 23:35
01:00 - 01:30
03:02 - 03:32
I found ITF in SCIENCE MODE; It remained locked for the whole shift.
No DMS Events to report
Guard Tour (UTC):
17:08 - 17:35
19:01 - 19:29
The ITF kept science mode all the shift.
Guard tours (UTC):
05:55 --> 06:30
08:00 --> 08:35
09:48 --> 10:33
11:55 --> 12:30
Nicola called me because the WI CO2 laser changed its state (see figure 1).
To address the change in the WI outer power, I first rotated the WI outer ring waveplate to recover the nominal power to be injected into the ITF (11.59 UTC-12.00.26 UTC ) (ITF in ‘adjusting’ mode). Then, I adjusted the WI chiller set point to return the laser to its usual operating point, in particular I decreased it by 0.01 degrees, from 19.02 to 19.01 (12.21.04 UTC). The BCK chiller set point has been changed accordingly. This action did not solve the problem; in fact, the direction turned out to be wrong.
Thus, at 12.41 UTC, I increased the chiller set point from 19.01 to 19.03 degrees.
Finally, the standard situation has been restored (see fig.2).
Upon arrival I found ITF in Science Mode (LN3_SQZ).
At 09:12 UTC ITF unlocked due to an earthquake in Japan with Mag 6.5 (fig2). L1, H1 already down.
ITF relocked, Science Mode back at 10:16 UTC.
ISC
The DMS reports a mistuning of B4_56MHz (LSC_SSFS_Err_DPHI) since few days. To be check. Plot (1). ISC expert present onsite informed.
TCS
After the relock the DMS notified a Power_DAS_OUT_is_out_of_range for WI CO2 and Beam_is_inside_ITF for CH PICKOFF (plot3). I alerted TCS OnCall.
At 11:59 and at 12:57 UTC I set in ADJUSTING for two minutes in order to allow Nardecchia to turn the WI plate and the WI chiller temperature setpoint.
Guard Tour (UTC)
05:03 - 05:29
07:01 - 07:27
10:56 - 11:24
Upon arrival I found ITF just relocked and in Science Mode.
Unlocks:
- 23:19 UTC (?). Relocked at first attempt.
- 01:32 UTC probably due to the seismic activity in Taiwan. Relockeed at second attempt at 03:36 UTC.
Guard Tour (UTC)
21:06 - 21:34
23:08 - 23:34
01:04 - 01:30
03:15 - 03:45
Upon arrival I found ITF relocking after the earthquake.
Relocking in progress but ITF very noisy due to the high wind and sea activity increasing (plot3). The bad weather conditions prevent a stable lock for the whole shift.
ALS
I noticed that after the intervention at WE on ALS laser amplifier, the Green Power and the IR_PSTAB are not in a good working point (plot2).
SBE
After one of the unlock events SNEB Horizontal loop opened. Properly closed.
TCS
During a relock the CH TCS_CO2..REL3 got stuck at 1. I act ON Up via TTL_CO2.
Guard Tour (UTC)
21:06 - 21:34
22:55 - 23:25
01:04 - 01:50
03:07 - 03:39
ISYS
(22-04-2024 04:00 - ) From remote
Status: On-going
Description: PSL_PMC_PZT out of range (plot1). Normally after an unlock it resets but not this night (?)
DMS comment: PMC_PZT_likely_to_drift_away.
Upon arrival I found ITF stable locked at LN3_SQZ (Science Mode).
17:09 UTC unlock for unknown reason (TBC).
18:19 UTC relock after the usual cross-alignment in ACQUIRE_DRMI.
18:30 / 19:02 UTC - CALIBRATED_DF_PCAL.
19:02 UTC - Science Mode set.
NO DMS Events Monitor to report
Guard Tour (UTC)
13:58 - 14:33
15:54 - 16:29
17:50 - 18:22
19:25 - 19:55
Upon arrival I found ITF stable in Science Mode (LN3_SQZ).
At 13:29 and at 13:33 UTC the SQZ unlocked, relocked at first attempt in both occasions (Science Mode set) but at third failure I contacted SQZ OnCall who checked on it. (Science Mode with ITF in LN3). The issue was related to the humidity level in Det Lab.
DET
13:39 UTC - DetMoni VPM process restarted under request of Gouaty.
At 14:24 UTC I set ADJUSTING in order to allow:
- ACS experts to fix the problem on DET Humidity. Activity concluded at 14:40 UTC. The steam producer was blocked. Soldani pointed out to me that the ENV DMS flag (ENV..SQZ_BOARD_HU) values are not "aligned" with the INF DMS flag (HVAC..DET_HU_IN).
- VAC experts to go at NE to check a CryoTrap valve. Activity concluded at 14:45 UTC.
- ALS Gosselin to reboot green ALS Laser but unfortunately the optic amplifier is not responding anymore. A replacement is planned for tomorrow morning -> AT NEXT UNLOCK WE ARE NOT ABLE TO RELOCK !
15:21 UTC - ITF again in Science Mode
CAL
In order to avoid the risk of ITF unlock, Was asked me to perform "only" 10 times the CALIBRATED_DF_DAILY.
19:00 UTC - ITF set in LN3 (CALIBRATION MODE) Also L1 and H1 DOWN.
- Calibrations started at 19:01 and concluded at 20:54 UTC
LN3_SQZ / Science Mode at 20:55 UTC.
Oncall events
SQZ
(19-04-2024 14:00 - 19-04-2024 14:25) From remote
Status: concluded
Description: The result of check of the SQZ failure was due to the humidity in DET area ENV..SQZ_BOARD_HU.
ACS
(19-04-2024 14:25 - 19-04-2024 14:45) On site
Status: concluded
Description: Det Humidity intervention by ACS Expert OnCall, steam producer restarted. Humidity values again in threshold.
ALS
(19-04-2024 13:00 - 19-04-2024 15:20) From remote
Status: concluded
Description: The issue was at level of optic amplifier. Aymeric contacted ISYS OnCall expert (C. DeRossi) who will come on site tomorrow morning to replace the device at WE with a spare one.
TCS
(19-04-2024 16:36 - 19-04-2024 17:00) From remote
Status: concluded
TCS_WI_CO2_PWROUT out of range. Nardecchia asked me to change the DMS thresholds.
Guard Tour (UTC)
17:01 - 17:40
19:01 - 19:28
We have updated the Shutter flag in the DMS (configuration of process DetMoni). In case the slow shutter motor remains stuck after step 109 of the lock acquisition, a red flag will be displayed.
We plan to further update the flag logic at the next maintenance to also perform some checks during the lock acquisition after the shutter closure.
Upon arrival I found ITF in Science Mode.
ITF stable locked at LN3_SQZ (Science Mode) till 11:31 UTC when it unlocked due to the bad weather conditions. High wind activity.
According with Gouaty I set ITF in SINGLE_BOUNCE_NI in order to allow Romain on his investigation on OMC shutter and B1_PD3 noise.
TROUBLESHOOTING MODE set from 11:33 to 12:56 UTC. Activity concluded. Relock in progress...
NO DMS events Monitor to report
The unlock due to bad weather was not associated to the limited range of the actuators, as happened other times. When the wind increased and reached 50 km/h in a few minutes, WE_MIR_CORR increased, but not enough to stay so close to the saturation (fig 1). The unlock happened during a clear drop of the sidebands, at the end of which some longitudinal loop was not very stable anymore (fig 2). The drop was associated to a misalignment of BS TY (fig 3) and PR_TY (fig 4). It is not clear to me which was the first which began to move, and what was the cause of this motion.
Upon arrival I found ITF in Science Mode (LN3_SQZ).
At ~ 08:50 UTC the vertical position loop of SWEB opened. I set Adjusting Mode and closed it but it opened again after few minutes. I tried to act on MotSwitch but it was not working. I contacted SBE OnCall but the issue was related to the old version of MotSwitch VPM config file. I restarted the MotSwitch process and act on MotSwitch Olwin tool to recover the vertical position of the bench. I made a check for all SBE MotSwitch processes are running with the v4r4 version.
Science Mode set again at 09:45 UTC.
No DMS Events Monitor to report
Oncall eventsDET
(16-04-2024 19:30 - 23:15) From remote
Status: On-going
Description: unlock at DC_READOUT step
SBE
(17-04-2024 10:00 - 17-04-2024 10:30) Operator on site with expert from remote
Status: Ended
Description: SWEB vertical loop open.
Problem fixed thanks to Carbognani who investigated on MotSwitch VPM config file process.
Upon arrival, I found ITF locked at LN3_SQZ, in CALIBRATION Mode.
At 23:04 UTC the calibration of Sensitivity concluded Science Mode set.
ITF unlocked once at 01:09 UTC (TBC). It relocked easily at the first attempt.
ITF in Science Mode for the rest of the shift.
Guard Tours (UTC)
21:38 - 22:08
23:25 - 23:54
01:45 - 02:16
03:50 - 04:20
I've noticed that the ITF unlocks some 20 minutes after reaching LOW_NOISE_3_SQZ, due to a 5 Hz oscillation visible mostly on PR_TX (Fig.1).
I have decreased PR_TX gain from 3.3 to 2.2 in CARM_NULL_1F, this gain is not changed for the rest of the lock acquisition. This does not seem to affect the lock acquisition in any undesirable way.
This change in gain might be motivated by a slow increase in sidebands power (Fig.2), after the drift in the WI DAS power (63940) around 19:00:00 UTC.
Upon arrival I found ITF relocking. The B1p spot was miscentered respect to the camera, so I moved with the fixed setpoint up to CARM_MC_IR, then I engaged again the floating setpoint.
ITF reached LN3_SQZ but at 22:00 UTC we discovered a problem with TCS_WI_CO2_PWROUT. I contacted TCS expert who fixed the problem (ADJUSTING MODE set).
After a couple of locking attempts ITF locked again in a stable way.
SCIENCE_MODE set at 23:37 UTC.
ITF unlocked 3 times during ACQUIRE_LN1 (TBC) so I decided to wait a bit in DC_READOUT. LN3_SQZ acheived at 04:30 UTC.
SCIENCE_MODE set at 04:31 UTC.
Guard Tours (UTC time)
21:06 - 21:35
23:01 - 23:29
01:02 - 01:31
03:02 - 03:31
DAQ
NewtonNoise
02"15 UTC - NNC_CEB_LEV2 process crashed. I tried to restart it via VPM without success:
Log:
FATAL..-Device V1:NN_CEB_INF_Z_209> No response after IP configuration
WARNING-Device V1:NN_CEB_ACC_Z_226> Writing invalid data with comment INFO_ZEROS, INFO_DATA_INVALID, INNO_DATA_TOO_SHORT, INNO_DATA_LENGTH, SERVICE_TIME_INTERVAL_HIGH, SERVICE_TIME_INTERVAL_SKIPPED, SERVICE_DATA_NO_REQ_LAST_SEC, SERVICE_DATA_OVERWRITTEN,
WARNING-Device V1:NN_CEB_ACC_Z_217> Writing valid data with comment SERVICE_TIME_INTERVAL_HIGH, SERVICE_TIME_INTERVAL_SKIPPED, SERVICE_DATA_NO_REQ_LAST_SEC, SERVICE_DATA_OVERWRITTEN,
TCS
(11-04-2024 21:55 - 11-04-2024 22:20) From remote
Status: Ended
Description: Power_DAS_OUT_is_out_of_range
TCS_WI_CO2_PWROUT at 3.6
The lock acquisition failures in fact happened around the transition to LN1, but the transition itself was fine, with no 1.8 Hz oscillation or other evident issue. The spikes visible in the corrections are normal: they are due to NI/WI relays switch.
he third unlock occurred before the transition.
Around 22.00 UTC, the operator called me because the DMS flag for TCS_WI_CO2_PWROUT became red.
Once connected, I noticed that the related signal fell down, below the threshold, because of a change in the WI main CO2 laser state: see all WI CO2 related channels in Fig. 1.
I recovered the correct value of the power injected inside the ITF with the WI DAS outer ring (see TCS_WI_CO2_POWER_CH_PICKOFF channel) by rotating the relative waveplate (ending around 22.20 UTC), to restore the ITF thermal compensation condition for locking.
The flag related to the channel TCS_WI_CO2_PWROUT would remain red: I’ve asked the operator to temporarily shelve it; the WI CO2 main laser condition should be checked again tomorrow morning.
Upon arrival I found ITF DOWN and commissioners working on the oscillation at 1.8 Hz. Activity interrupted, but not completely concluded, at 23:30 UTC.
Locking problems are not yet fixed and ITF is very unstable and noisy.
ITF unlocked at several steps at different steps (TBC) without reaching LN3.
At 03:18 UTC, after an unlock, I tried the way to Reload Filters in LSC_Acl and ITF reached LN2 for the first time but unfortunately the GALVO B2 QD2 remained Open, I closed it via VPM but after one minute ITF unlocked again.
Relocking in progress...
Guard Tours (UTC time)
21:12 - 21:40
23:05 - 23:25
01:00 - 01:34
03:10 - 03:43
Upon arrival I found ITF in Science Mode, stable locked in LN3_SQZ.
At 07:42 UTC ITF unlocked due to th PR trip (plot1).
At 9:00 the SWEB DAQ Box got stuck I tried to restart the VPM process but the issue remained. A conseguence of this failure was on Hrec_Range_BNS and HOFT_Bias that drop to 0.
TROUBLESHOOTING MODE set.
I contacted Masserot who fixed the problem at 11:00 UTC.
Consequentially of to this problem, there are some missing data (gray flags on DMS) and SWEB_dbox_rack still down. Tomorrow, during the maintenance, the electronic crew will replace the old DAQbox with a new one.
SWEB_SBE loops opened and properly closed with ITF locked at LN3_SQZ.
After several unlocks, ITF was back in LN3 and LN3_SQZ at 12:30 UTC.
COMMISSIONING MODE set.
According with Sorrentino we decided to allow Boschi to work on PR board in CEB from ~13:00 UTC.
SQZ
during a slot out of Science, Vardaro profit to fix the SQZ_INJ (AA Mirror1, 2 and Move M32).
DAQ
(08-04-2024 09:30 - 08-04-2024 11:00) From remote
Status: Ended
Description: SWEB DAQ Box failure.
Since saturday night the SQZ_MIRROR1_X was out of range, correction about 9.5 V. I profit to discharge it by moving SQB1_M32 picomotor in H and V.
I noticed that the good direction to increase SQB1_MIRROR1_X voltage is to move SQB1_M23_H toward negative direction (steps of 100) and SQB1_M23_V toward positive direction.
After that the SQZ_MIRROR1_X is around 5V. To be checked in the next days
Upon arrival I found ITF in Science Mode (LN3_SQZ).
At 07:16 UTC ITF unlocked due to a PR trip (plot attached).
Lock acquisition restarted:
07:28 UTC - Unlock at CARM_NULL_1F (?). After the unlock, in DOWN state, I reloaded filters of LSC_Acl as suggest by Bersanetti.
08:14 UTC - Unlock at LN2 (?)
ITF back in Science Mode at 08:55 UTC.
Guard tours (UTC):
06:00 --> 06:35
08:05 --> 08:55
10:00 --> 10:35
11:52 --> 12:26
Upon arrival I found ITF locked at LN3_SQZ with SQZ Injections activity still ongoing (Commissioning).
At 22:00 UTC the activity concluded and I set Science Mode.
Unfortunately at 00:12 UTC ITF unlocked for a strong earthquake occurred at Taiwan (Mag 7.4).
The seismic events unlocked the Inj system, shook all suspensions and opened the loops of SDB2, SIB2, EIB, SWEB, SNEB for 2 hours. All loops properly closed.
I started to relock only at 02:30 UTC.
LN3_SQZ reached at 03:28 UTC. Science Mode set.
04:30 UTC unlocked due to a PR trip (plot attached). Relocking in progress...
Guard tours (UTC):
21:10 --> 21:40
23:15 --> 23:45
01:00 --> 01:30
03:10 --> 03:40
DAQ
00:00 UTC - FbTrend_10800 process crashed / restarted via VPM.
